Skip to content

Increase database healthcheck tolerance for production environments#51

Merged
PhantomDave merged 2 commits intomainfrom
copilot/fix-database-healthcheck-issue
Nov 14, 2025
Merged

Increase database healthcheck tolerance for production environments#51
PhantomDave merged 2 commits intomainfrom
copilot/fix-database-healthcheck-issue

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Nov 14, 2025

Production deployments fail with "container banktracking-database is unhealthy" due to insufficient healthcheck grace period and retry limits in resource-constrained environments.

Changes

  • Healthcheck timing: Increased start_period from 10s to 30s to accommodate initial volume initialization
  • Retry limits: Increased retries from 5 to 10, extending total retry window from 50s to 100s
  • Exit code handling: Added explicit || exit 1 to healthcheck command for proper failure signaling

Applied consistently across compose.yaml, compose.dev.yaml, and compose.local.yaml.

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U bankuser -d bankdb || exit 1"]
  interval: 10s
  timeout: 5s
  retries: 10        # was: 5
  start_period: 30s  # was: 10s
Original prompt

The job is failing because the container banktracking-database is unhealthy and cannot start, leading to a dependency failure. The logs show:

  • "Container banktracking-database Error"
  • "dependency failed to start: container banktracking-database is unhealthy"

Solution Steps

  1. Check the Docker Compose or workflow configuration for the banktracking-database service.
  2. Verify the healthcheck settings for the database container; unhealthy status usually means the healthcheck is failing.
  3. Ensure that environment variables (e.g., database passwords, ports) are set correctly and any required secrets are available.
  4. Inspect the database image being used. Make sure it's a valid and up-to-date image.
  5. If there are initialization scripts, make sure they don't fail. Errors in database schema initialization or permissions often cause healthcheck failures.

Code Example: If you are using a docker-compose.yml and a healthcheck is set, consider adding logging to the healthcheck command to debug further. For example:

yaml
services:
banktracking-database:
image: postgres:15
environment:
POSTGRES_USER: example
POSTGRES_PASSWORD: examplepw
healthcheck:
test: ["CMD-SHELL", "pg_isready -U example"]
interval: 10s
timeout: 5s
retries: 5

Make sure to match the POSTGRES_USER and POSTGRES_PASSWORD with your secrets or environment variables.

Next Steps:

  • Check .github/workflows/deploy-tailscale.yml and your database service definition for errors.
  • If the healthcheck is incorrect or too strict, adjust it.
  • Review logs from the failing database container ("docker logs banktracking-database" if running locally) to spot underlying errors.

If you can share the database service definition or any custom healthcheck, I can provide more focused suggestions.

Deployement is failing


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: PhantomDave <34485699+PhantomDave@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix unhealthy status of banktracking-database container Increase database healthcheck tolerance for production environments Nov 14, 2025
Copilot AI requested a review from PhantomDave November 14, 2025 22:23
@PhantomDave PhantomDave marked this pull request as ready for review November 14, 2025 22:36
Copilot AI review requested due to automatic review settings November 14, 2025 22:36
@PhantomDave PhantomDave merged commit 2c90485 into main Nov 14, 2025
10 of 13 checks passed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses production deployment failures caused by the database container being marked unhealthy before it fully initializes. The changes extend healthcheck tolerance across all Docker Compose configurations to accommodate resource-constrained production environments.

Key Changes

  • Extended healthcheck grace period from 10s to 30s for initial volume initialization
  • Doubled retry attempts from 5 to 10, extending the total retry window from 50s to 100s
  • Added explicit exit code handling (|| exit 1) to align with existing healthcheck patterns used by other services

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
compose.yaml Updated production database healthcheck with increased tolerance and explicit failure signaling
compose.dev.yaml Applied consistent healthcheck improvements for development environment
compose.local.yaml Applied consistent healthcheck improvements for local testing environment

@PhantomDave PhantomDave deleted the copilot/fix-database-healthcheck-issue branch November 15, 2025 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants